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(57) Abstract: Systems and methods for endoscopic procedures employ a first 
technique to ensure initial correct alignment of an endoscope (100) with a de- 
sired target (10). A reference image (51) is then acquired from an imaging 
arrangement associated with the endoscope. During a subsequent stage of 
the procedure, tracking of the endoscope position relative to the target is per- 
formed partially or entirely by imageObased tracking by comparing features in 
real-time video image (52) produced by imaging arrangement with features in 
the reference image (51). The feature comparison may be performed visually 
by a user, or may be automated to offer more specific corrective suggestions 
to the user. 
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System and Method for Image-Based Aligmnent of an Endoscope 

FIELD AND BACKGROUND OF THE INVENTION 

The present invention relates to endoscopy techniques and, in particular, it 
concerns a system and method for image-based alignment of an endoscope during 
5 at least part of an endoscopic procedure. 

The present invention will be exemplified in the context of a system as 
described in the co-assigned PCT application published as WO 03/086498 entitled 
"Endoscope Structure and Techniques for Navigation in Branched Structure" to 
Gilboa^ which is hereby incorporated by reference in its entirety. The 

10 aforementioned patent application discloses a method and apparatus in which a thin 
locatable guide, enveloped by a sheath, is used to navigate a bronchoscopic tool to 
a target location within the lung, aimed in particular to deliver treatments to the 
lung periphery beyond the bronchoscope's own reach. The coordinates of the target 
are predetermined based upon three-dimensional CT data. A location sensor is 

15 incorporated at the locatable guide's tip. The enveloped guide is inserted into the 
lung via the working channel of a bronchoscope. First, the bronchoscope's tip is 
directed to the furthest reachable location in the direction of the target. Next, the 
guide is advanced beyond the tip of the bronchoscope towards the designated 
target, based on the combination of the CT data and the position of the guide's tip 

20 as measured in body coordinates. When the guide's tip at the target, the guide is 
withdrawn, freeing the enveloping sheath for insertion a bronchoscopic tool. In 
order to prevent the distal end portion of the sheath from sliding away from the 
target, the sheath is locked to the bronchoscope's body and the bronchoscope itself 
. is held steadily to prevent it from slipping further into the lungs or outwards. 

25 Because the airways in the periphery of the lung are narrow, approximately in the 
same dimensions as the sheath, sideways movements are extremely limited. 

The above system may also be used to navigate the tip of the bronchoscope 
to a target located inside the main bronchus and not only to targets in the periphery 
of the lungs. Although for such centrally-located target the physician has direct 

1 
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visualization of the scene in front of the bronchoscope, it is not always sufficient 
for visually identifying the designated targets, since many of these targets are 
hidden in the tissue outside the airways. Hence, it is a benefit to combine the CT 
data into the navigational aids also for targets inside the main bronchus, where the 
5 bronchoscope can reach and direct vision exists, but yet the target itself is hidden. 

When using the navigation system for navigating the tip of the bronchoscope 
itself, many of the mechanical features of the locatable guide described in 
WO 03/086498 are not needed. Specifically, the steerability of the guide is not 
needed, and the enveloping sheath is also not needed. However the principle of 

10 using a separate locatable guide having a location sensor at its tip and being 
inserted into the working channel of a regular bronchoscope actually changes the 
bronchoscope from a non-locatable bronchoscope to a locatable bronchoscope, 
thereby offering major advantages as will become clear. 

As in the prior art apparatus, the locatable guide can be inserted into and 

15 withdrawn from the bronchoscope's working channel as needed. Unlike the 
periphery of the lung, the central airways are much wider than the bronchoscope. 
As a consequence, when the tip of the bronchoscope is on target, it can move 
sideways in addition to sliding in and out. Therefore stabilizing the bronchoscope's 
tip during treatment is a three dimensional task, involving the operation of the 

20 steering ability of the bronchoscope. An example for the importance for 
maintaining the location of the bronchoscope's tip at the designated target during 
the insertion of the bronchoscopic tool is the use of the Transbronchial Histology 
Needle, by which a needle is guided towards a target such as a lymph node which 
neighbors the main bronchus from the outside and thus is invisible to the 

25 bronchoscope image but its coordinates are known from the CT data. Any mistake 
in directing the needle may result not only in failure of the procedure, but worse, in 
causiag damage to vital organs such as the aorta or other major blood vessels. 

In principle, the same methods as presented in WO 03/086498 may be used 
in the context of the major airways. Specifically, by using the location of the tip of 

30 the bronchoscope as measured by the location measurement sensor, a directing 

2 
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display is produced corresponding to a simulation or schematic diagram of the view 
jBrom the distal tip of the guide, which is based on the relative location of the target 
versus the position of the tip of the guide in six degrees of iBreedom, In the central 
airways, this view is supplemented by the direct video image from the 
5 bronchoscope imaging arrangement Based on these two displays, the physician 
brings the tip of the bronchoscope to the target. When the tip of the bronchoscope is 
correctly aligned with and adjacent to the target (Figure 7)^ the guide with the 
location sensor is withdrawn (as shown in Figure 8), thereby freeing the 
bronchoscope's working channel for insertion a bronchoscopic tool (Figure 9a). 

10 Once the locatable guide is released, the directing display can no longer function 
for directing the tip to target. Instead, the physician has to hold the bronchoscope as 
steadily as possible during withdrawal of the guide and the insertion of the tool. If 
the bronchoscope slips from the target location (for example, as shown in Figure 
96), the physician may notice the change of position in the video image, but has no 

15 effective tool available to help him return the tip of the bronchoscope reliably to the 
desired target (other than reinserting the guide and repeating the navigation 
process). 

Hence, it would be of benefit to have a metliod and corresponding system 
for confirming correct alignment of the tip of an endoscope after removal of a 
20 locatable guide used to achieve initial alignment, particularly for procedures 
involving a target which is obscured from view. 

SUMMARY OF THE INVENTION 

The present invention is a system and method for image-based aUgnment of 
an endoscope. 

25 According to the teachings of the present invention there is provided, a 

method for confirming correct alignment of a distal end of an endoscope including 
an imaging arrangement during an endoscopic procedure, the method comprising: 
(a) positioning the distal end of the endoscope adjacent to a target location and 
capturing a reference image using the imaging arrangement; (b) sensing a real-time 
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video image using the imaging arrangement; and (c) comparing features of the real- 
time video image with the reference image to confirm correct aligmnent of the 
endoscope. 

According to a further feature of the present invention^ the step of 
5 positioning employs a target location identified in three-dimensional image data of 
a region of a body to be treated. 

According to a further feature of the present invention^ the three-dimensional 
image data is derived from an imaging technique selected from: computerized 
tomography; magnetic resonance imaging; positron emission tomography; and 
10 ultrasound. 

According to a further feature of the present invention, the step of 
positioning employs a position sensor associated with the distal end of the 
endoscope, the position sensor being part of a position measuring system. 

According to a further feature of the present invention, the step of 
15 positioning is performed by comparing the position of the distal end of the 
endoscope as measured by the position measuring system and the target location as 
identified in the image data. 

According to a further feature of the present invention, the position sensor is 
part of an elongated element deployed within a working channel of the endoscope, 
20 and wherein the elongated element is withdrawn from the working channel prior to 
the comparing- 

According to a further feature of the present invention, the target location is 
not visible in the reference image. 

According to a further feature of the present invention, the reference image 
25 and the real-time video image are displayed simultaneously to facilitate 
performance of the comparing features visually by a user. 

According to a further feature of the present invention, the comparing 
includes co-processing the reference image and at least one frame from the real- 
time video to determine a measure of mismatch, the method further comprising 
30 generating an alarm signal if the measure of mismatch exceeds a predefined value. 
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According to a further feature of the present invention, the comparing 
includes co-processing the reference image and at least one frame from the real- 
time video to determine a displacement correction required to compensate for 
erroneovis movement of the endoscope, the method further comprising generating a 
5 display indicative to a user of the displacement correction required to compensate 
for the erroneous movement of the endoscope. 

According to a further feature of the present invention, the comparing 
includes co-processing the reference image and at least one frame from the real- 
time video to determine a transformation relating the real-time video frame to the 
10 reference image, the method further comprising generating a display corresponding 
to the real-time video with addition of an indication of a target location, position of 
the indication being derived at least in part by use of the transformation. 

According to a further feature of the present invention, the endoscope is a 
bronchoscope. 

15 There is also provided according to tiie teachings of the present invention, a 

system for ensuring correct alignment of an endoscope during performance of an 
endoscopic procedure, the system comprising: (a) an endoscope having a distal end 
for insertion into a body; (b) an imaging arrangement associated with the 
. endoscope and configured to generate a real-time video image of a region beyond 

20 the distal end; and (c) a processing system associated with the imaging arrangement 
and configured to: (i) in an initial state of alignment with a target location, derive 
from the imaging arrangement a reference image corresponding to correct 
alignment with the target location, (ii) derive from the unaging arrangement real- 
time images of the region beyond the distal end, and (iii) co-process the reference 

25 image and the real-time images to determine a current alignment status of the 
endoscope with the target location. 

According to a further feature of the present invention, the processing 
system is configured to co-process the reference image and the real-time images to 
determine a measure of mismatch, the processing system further generating an 

30 alarm signal if the measure of mismatch exceeds a predefined value. 
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According to a further feature of the present invention, there is also provided 
a display for displaying at least the real-time images to a user, wherein the 
processing system is configvired to co-process the reference image and the real-time 
images to determine a displacement correction required to compensate for 
5 erroneous movement of the endoscope, the processing system further generating an 
indication on the display indicative to a user of the displacement correction 
required to compensate for the erroneous movement of the endoscope. 

According to a further feature of the present invention, there is also provided 
a display for displaying at least the real-time images to a user, wherein the 

10 processing system is configured to co-process the reference image and the real-time 
images to determine a transformation relating the real-time video frame to the 
reference image, the processing system further generating on the display an 
indication of a target location, position of the indication being derived at least in 
part by use of the transformation. 

15 According to a further feature of the present invention, there is also provided 

a position measuring system including a position sensor carried by an elongated 
element removably deployable along a working channel of the endoscope. 

There is also provided according to the teachings of the present invention, a 
method for facilitating performance of an endoscopic procedure on a target which 

20 is obscured from view by an imaging arrangement of an endoscope, the method 
comprising: (a) generating real-time video from the imaging arrangement of the 
endoscope; (b) determining a position of the target and a position of a distal end of 
the endoscope; (c) generating a display including the real-time video and a 
simulated view of the target correctly positioned within the real-time video; and 

25 (d) adjusting the display so as to maintain the target correctly positioned within the 
real-time video when the endoscope is moved. 

According to a further feature of the present invention, the adjusting 
includes comparing features from at least one frame of the real-time video with 
features from a reference image derived from the imaging arrangement during the 



6 
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Step of determining, thereby deriving a transformation relating the real-time video 
frame to the reference image. 

According to a further feature of the present invention, the determining a 
position of a distal end of the endoscope is performed using position measuring 
5 system including a position sensor carried by an elongated element removably 
deployable along a working channel of the endoscope. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is herein described, by way of example only, with reference to 
the accompanying drawings, wherein: 
10 Figure 1 is a video display of the target zone inside the bronchial tree when 

the tip of the bronchoscope is off target. 

Figure 2 is a simulated tip view corresponding to the position of Figure L 
Figure 3 is a video display of the target zone inside the bronchial tree when 
the tip of the bronchoscope is on the target. 
15 Figure 4 is a simulated tip view corresponding to the position of Figure 3 . 

Figure 5 is a display of a combination of a stored image and live video in the 
context of first preferred embodiment of the invention. 

Figure 6 is the display of a combination of a stored image and live video in 
the context of a second preferred embodiment of the invention. 
20 Figure 7 is a schematic side cross-sectional view showing the bronchoscope 

having been correctly aligned by use of a position measurement sensor with a target 
which is obscured from view. 

Figure 8 is^ a view similar to Figure 7 after removal of the position 
measurement sensor to free a working lumen of the bronchoscope. 
25 Figure 9a is a view similar to Figure 8 after insertion of a tool along the 

working lumen. 

Figure 9b is a view similar to Figure 9a after erroneous movement has 
disrupted alignment of the tool with the obscured target. 

7 
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Figure 10 is a schematic illustration of tiie components of a system, 
constructed aad operative according to the teachings of the present invention, for 
ensuring correct alignment of an endoscope during performance of an endoscopic 
procedure. 

5 DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The present invention is a system and method for image-based ahgnment of 
an endoscope. 

The principles and operation of systems and methods according to the 
present invention may be better \inderstood with reference to the drawings and the 

10 accompanying description. 

First in general terms, the present invention provides systems and methods 
for endoscopic procedures (exemplified herein with reference to bronchoscopic 
procedures) wherein a first technique is used to ensiare initial correct alignment of 
an endoscope with a desired target and a reference image is acquired from an 

15 imaging arrangement associated with the endoscope. Then, during a subsequent 
stage of the procedure, tracking of the endoscope position relative to the target is 
performed partially or entirely by image-based tracking by comparing features in 
the real-time video image produced by the imaging arrangement with features in 
the reference image. 

20 Thus, according to a first aspect of the present invention, a method for 

confiiming correct alignment of a distal end of an endoscope during an endoscopic 
procediH-e includes: positioning the distal end of the endoscope adjacent to a target 
location and capturing a reference image using the imaging arrangement; sensing a 
real-time video image using the imaging arrangement; and comparing features of 

25 the real-time video image with the reference image to confirm correct alignment of 
the endoscope. 

It will immediately be appreciated that the present invention offers profound 
advantages, particularly for cases where the desired target is obscured from view 
(such as behind other tissue) or is not readily identifiable directly by visual 
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imaging. In such cases, navigation under video imaging alone is insufficient. 
Nevertheless, after use of a primary tracking system (such as that of the 
aforementioned WO 03/086498) to achieve initial aligrmient, use of feature-based 
optical tracking based on features not necessarily belonging to the target frees the 
5 system from subsequent dependence on the primary tracking system, thereby 
allowing removal of the position measurement probe and/or rendering navigation 
more robust and reliable in the face of disturbances such as movement of the 
patient's body or the like. These and other advantages of the present invention will 
become clearer from the subsequent description. 

10 Referring now to the drawings, Figure 10 shows schematically a preferred 

implementation of a system, constructed and operative according to the teachings 
of the present invention, for implementing the methods of the present invention. 
For one set of preferred implementations of the method of the present invention, the 
system is substantially similar to that described in WO 03/086498 with certain 

15 changes to the display and/or image processing systems, as will be described 
below. Thus, a position measurement sensor 101 and video sensor 102 are 
incorporated in tiiie distal tip of bronchoscope 100. An electro-magnetic tracking 
system induces electro-magnetic fields from antennae 107, senses the signals from 
the location sensor 101 and determines the position of the tip of the bronchoscope 

20 in six degrees of freedom. A processing system 108 gathers lhat position 
information together with the video image from the tip of the bronchoscope as 
produced by the video camera 104. The processing system may display to the 
physician live video, captured images and simulated views on a display screen 110. 
Further details of a particularly preferred position measuring system for 

25 measxaring position in six degrees-of-freedom may be found ia U.S. Patent 
No. 6,188,355 and PCT Application Publication Nos. WO 00/10456 and 
WO 01/67035. Most preferably, at least one, and preferably three, reference sensors 
(not shown) are also attached to the chest of the patient and their 6 DOF 
coordinates sent to processing system 108 where they are used to calculate the 

30 patient coordinate frame of reference. 
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It should be noted in this context liiat the term "position sensor" is used 
herein in the description and claims to refer to any element which can be associated 
permanently or temporarily with an object and functions together with other 
components of a position measuring system to determine the position and/or 
5 attitude of the object It should be appreciated that the terminology does not 
necessarily imply that the position sensor itself is capable of any measurement 
alone. Nor does this terminology imply any particular function of the position 
sensor, such that the "sensor" may be a transmitter, a receiver or any other element 
which functions as part of a position measuring system, depending upon the 

10 technology employed. In all such cases, the element is referred to as a "position 
sensor" since its presence associated with the object allows sensing by the system 
of the object's position. 

Although described herein with reference to a non-limiting preferred 
implementation employing a bronchoscope, it should be noted that the present 

15 invention is equally applicable to substantially any intra-body endoscopic 
procedure. 

As in the aforementioned WO 03/086498, the location of the desired target 
within the body is preferably determined in an offline preparation session prior to 
the procedure in which the target is identified in three-dimensional image data of a 

20 region of a body to be treated. The three-dimensional image data is preferably 
derived from an imaging technique selected from: computerized tomography; 
magnetic resonance imaging; positron emission tomography; and ultrasound. Most 
commonly, computerized tomography ("CT") data is used. Then, after suitable 
calibration to register the position measurement system coordinates with the CT 

25 data, a simulated tip view or other visual navigation aids as described in 
WO 03/086498 are used to guide the bronchoscope into alignment with the target. 
These navigation aids are based on comparing the position of the distal end of the 
endoscope as measured by the position measuring system and the target location as 
identified in the image data. Then, according to one particularly preferred set of 



10 



wo 2005/102175 



PCT/IL2005/000452 



embodiments, the position sensor is withdrawn from a working channel of the 
endoscope as part of an elongated element 

Figures 1-4 illustrate schematically examples the displays which are 
preferably available to the physician during initial alignment of the bronchoscope. 
Specifically, Figure 1 shows an example of a target area. The target is obscured 
from view, being located behind the tissue of a bifurcation inside the bronchus. The 
target 10, marked in a broken line, is not visible in the video image. In one 
embodiment of the invention, it is not marked in the video display at all. In another 
preferred embodiment it's the target's location, as calculated by the processing 
system 108, is displayed in the video display by an artificial mark such as a line, a 
point, a broken line, a colored area, a tiiree dimensional entity or a combination of 
any of the above. 

In Figure 1, the tip of the bronchoscope is shown to be positioned off the 
direction of the target Figure 2 is the tip view used for directing to the target The 
direction of the target relative to the tip is presented by dot 20 marking the target 
and arrow 22 aiming from the tip to the target. The target may also be presented in 
a simulated view of the boundary of the actual target lesion as calculated from the 
CT data or by a colored area or by three-dimensional entity or by any combination 
of the above. According to the example, the tip should be deflecting in the 12 
o'clock direction in order to be on target Figures 3 and 4 show the same scenario 
when the tip is on target. According to the prior art described above, after achieving 
alignment with the target as shown, the physician has to try to hold the 
bronchoscope steady while withdrawing the locatable guide and inserting a tool 
along the lumen. According to the present invention, before the guide is withdrawn, 
the image as shown in Figure 3 is captured and stored in a memory device of 
processing system 108. Now the system has two sources of images to control the 
location of the bronchoscope's distal tip, a real-time live video image 52 and a 
captured video image 51 where the tip was located at the desired target location, as 
shown in Figure 5. 
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The present invention may be implemented in a number of different 
embodiments with different degrees of sophistication as to how the comparison 
between features of the real-time video and the reference image is performed. 
According to a first basic embodiment, the reference image 51 and the real-time 

5 video image 52 are displayed simultaneously on display device 1 10 as illustrated in 
Figure 5, thereby facilitating visual comparison of the image features by a user. In 
this case, the physician himself compares the two images and decides whether the 
bronchoscope is located in the required location, and if not, in what direction the tip 
of the bronchoscope should be deflected. 

10 In more sophisticated embodiments, tiie system preferably co-processes the 

reference image and the real-time images to determine a current alignment status of 
the endoscope with the target location. Thus, processing system 108 is configiired 
to: derive firom the imaging arrangement of the endoscope, in an iaitial state of 
alignment with a target location, a reference image corresponding to correct 

15 aligmnent with the target location; derive from the imaging arrangement real-time 
images of the region beyond the distal end, and co-process the reference image and 
the real-time images to determine a current aligmnent status of the endoscope with 
the target location. 

Here too, the co-processing may be implemented at various different levels 
20 of sophistication. In a simplest case, a correlation between the reference image and 
the cvircnt video image may offer a measure of mismatch. The user can then 
empirically adjust the position of the bronchoscope tip to maximize the correlation 
(minimize the mismatch), thereby returning to the correct position. Application of a 
threshold to the measure of mismatch may be used to activate an alarm signal. 
25 In more preferred implementations, the system tracks features or regions 

from the reference unage in. the video image to provide more specific indications to 
the user of the required correction for any erroneous movement of the 
bronchoscope off target. For small-scale lateral displacements, this may be 
implemented simply by correlating a central sub-window 56 of reference image 51 
30 centered on target location 55 with a corresponding sized sliding window (i.e., at 



wo 2005/102175 



PCT/IL2005/000452 



multiple different positions) in the real-time video to find the best match, thereby 
identifying the position of the target sub-window in the real-time video image. 

At a next level of sophistication, the tracking may also allow for scaling 
and/or rotation of the sub-window. This allows the system to maintain target 
5 tracking during rotation, as well as small-scale advancing or withdrawal, of the 
bronchoscope. A further level of sophistication may employ planar transformations 
such as affine transformations which approximate the distortions caused by viewing 
a surface from different viewing angles. 

At the top end of the range of sophistication in the tracking algorithms are 

10 tracking techniques based on three-dimensional modeling of the viewed scene and 
reconstruction of the camera path. Such techniques, often referred to as "Structure 
from Motion", are well developed in the field of optical tracking and computer 
vision, and allow reconstruction of three-dimensional models from a single moving 
camera. Details of processing techniques for implementing structure from motion 

15 may be found in papers from the Robotics Research Group in ttie Department of 
Engineering Science, Oxford University (UK) such as "Automatic Camera 
Tracking" by Andrew W. Fitzgibbon et al. Video Registration (2003) and "Feature 
Based Methods for Structure and Motion Estimation" by P.H.S. Torr et al. Vision 
Algorithms: Theorv and Practice (2000), both available from 

20 http://www.robots>ox.ac.uk/ . 

In the present application, structure-from-motion processing can be greatly 
simplified by the use of model data based on CT data or the like. Thus, for 
example, given that the initial reference image is taken from a known position as 
estabUshed by the primary alignment system, a "depth" (i.e., camera-to-surface 

25 distance) associated with each pixel of the reference unage can be derived directly 
from CT data, thereby providing an initial three-dimensional model from which 
processing can begin. This approach has advantages of robustness under significant 
changes of view, and even where there is no overlap between the current real-time 
video field of view and the reference image. 



13 
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In any or all of the above-mentioned tracking techniques, corrections are 
preferably made for geometrical distortions introduced by the optics of the imaging 
arrangement, as is known in the art. These corrections may be performed on the 
source images prior to implementing the tracking techniques, or may be 
5 incorporated into the tracking calculations themselves. 

The output from the system (and method) of the present invention may take 
a nimiber of forms. In a simplest case mentioned above, an alarm may be sounded 
if a measure of mismatch between the current video and the reference image 
mdicates that the bronchoscope has wandered off target, and the measure of 
10 mismatch (or the correlation) may be displayed to the user or indicated by an audio 
signal to provide feedback indicative of "getting hotter" or "gettmg colder" with 
regard to alignment with the target. 

In more preferred implementations where features of the reference image are 
positively tracked within the real-time video, the processing system may generate 
15 an indication on the display indicative to a user of the displacement correction 
required to compensate for the eiToneous movement of the endoscope. This may 
take the form of an arrow or vector such as line 62 in Figure 6 which indicates the 
movement of the bronchoscope required to bring the center of the field of view into 
alignment with the target position illustrated as 61. Thus, Figure 6 corresponds to 
20 tlie display when the bronchoscope is in the position of Figure 9b. After performing 
the required corrective motion, the bronchoscope returns to the position of Figure 
9a and the live video 52 of Figure 6 would again appear similar to the reference 
image 51. 

Alternatively, or additionally, a transformation (2 or 3 dimensional) 
25 calculated by the processing system for relating the real-time video frame to the 
reference image may be used to determine the position of the obscured target 
within the real-time video image. The target location can tlien be designated in the 
real-time video image, for example, by the sub-window frame 58 or the marker 57 
as shown in Figure 5. 
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According to a most preferred option, which is believed to be of patentable 
significance in its own right, the present invention provides an augmented reality 
endoscopic display in which a simulated view of an obscured target is displayed m 
the context of the real-time video so that the target appears correctly positioned 
5 within the video image and moves so as to maintain the correct positioning of the 
target within the real-time video when the endoscope is moved. This augmented 
reality display allows the user to operate the endoscope in a fully intuitive manner 
as if the target were directly viewable via the video imaging arrangement of the 
endoscope. Thus, the user will see obscured target 10 of Figures 1 and 3 as if the 

1 0 tissue in front of the target was semi-transparent. 

In practical terms, the simulated view of the target used for the augmented 
reality display is preferably derived from three-dimensional imaging data such as 
CT in which the target tissue has been designated prior to the procedure. The target 
tissue volume is then preferably exported as a three-dimensional graphic object, or 

15 a closed body geometrical approximation to the tissue volume is generated. Then, 
during the procedure, information regarding the relative positions and orientations 
of the endoscope tip and the target tissue is used to determine the position, viewing 
angle and scaling factors which should be used to represent the target correctly in 
the real-time video image. The target is preferably indicated as a semi-transparent 

20 video overlay so that it appears as a ghost image without completely hiding the 
tissue actually viewed in the video image. Altematively, a dashed outline or any 
other suitable indication may be used. 

The augmented reality display is advantageous both durhig initial alignment 
of the endoscope with the target and during subsequent performance of a 

25 procedure. Most preferably, during a procedure, real-time adjustment of the 
simulated target image within the video image is performed on the basis of the 
optical tracking of the present invention. Where optical tracking is performed in 
two dimensions only, the adjustment of the target appearance will correspondingly 
be reduced to a two-dimensional manipulation. Where three-dimensional model 
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based tracking is used, full three-dimensional augmented reality fimctionality is 
preferably maintained. 

In each case, an alarm is preferably activated if the location error, i.e,, the 
misalignment of the real-tune video from the reference image, exceeds a predefined 
value, for example, the size of the target. The alarm may be an audio alarm and/or a 
visual alarm indication such as a flashing symbol or a color change of part or all of 
the display. An alami (which may be distinct from the off-target alarm) is 
preferably also generated if the tracking algorithm fails to maintain reliable 
tracking between the reference image and the current image. 

As mentioned earlier, the optical tracking of the present invention does not 
require the target location to be visible in the reference image. In fact, it should be 
noted that the reference image need not even include the direction to the target in 
its field of view. For example, if an endoscope is to be used in a procedure with a 
tool which generates images, takes samples or otherwise treats a region of tissue 
located laterally next to the distal end of the endoscope, the imaging arrangement of 
the endoscope will typically not offer a view of the tissue of the target, nor of any 
tissue which overlies the target. Nevertheless, once correct alignment of the 
endoscope has been achieved using the primary tracking system, the optical 
tracking of the present invention based on a reference image of the scene beyond 
title tip of the endoscope is effective to ensure correct aligrmient with the target, 
despite the fact that the target is outside the field of view. 

It will be appreciated that the above descriptions are intended only to serve 
as examples, and that many other embodiments are possible within the scope of the 
present invention as defined in the appended claims. 
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WHAT IS CLAIMED IS: 

1. A method for confirming correct alignment of a distal end of an 
endoscope including an imaging arrangement during an endoscopic procedure, Ihe 

method comprising: 

(a) positioning the distal end of the endoscope adjacent to a target 
location and capturing a reference image using the imaging 
arrangement; 

(b) sensing a real-time video image using the unaging arrangement; and 

(c) comparing features of the real-time video image with the reference 
image to confirm correct alignment of the endoscope. 

2. The method of claim 1, wherein said step of positioning employs a 
target location identified in three-dimensional image data of a region of a body to 
be treated. 

3 . The method of claim 2, wherein said three-dimensional image data is 
derived from an imaging technique selected from: computerized tomography; 
magnetic resonance unaging; positron emission tomography; and ultrasound. 

4. The method of claim 2, wherein said step of positioning employs a 
position sensor associated with the distal end of the endoscope, said position sensor 
being part of a position measuring system. 

5. The method of claim 4, wherein said step of positioning is performed 
by comparing the position of the distal end of the endoscope as measured by said 
position measuring system and the target location as identified in said image data. 

6. The method of claun 4, wherein said position sensor is part of an 
elongated element deployed within a working channel of the endoscope, and 
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wherein said elongated element is withdrawn from the working channel prior to 
said comparing. 

7. The method of claim 1, wherein said target location is not visible in 
said reference image. 

8. The method of claim 1 , further comprising displaying simultaneously 
said reference image and said real-time video image to facilitate performance of 
said comparing features visually by a user. 

9. The method of claim 1, wherein said comparing includes co- 
processing said reference image and at least one frame from said real-time video to 
determine a measure of mismatch, the method further comprising generating an 
alarm signal if said measure of mismatch exceeds a predefined value. 

10. The method of claim 1, wherein said comparing includes co- 
processing said reference image and at least one frame from said real-time video to 
determine a displacement correction required to compensate for erroneous 
movement of the endoscope, the method fiirther comprising generating a display 
indicative to a user of said displacement correction required to compensate for the 
erroneous movement of the endoscope. 

11. The method of claim 1, wherein said comparing includes co- 
processing said reference image and at least one frame from said real-time video to 
determine a transformation relating said real-time video frame to said reference 
image, the method further comprising generating a display corresponding to said 
real-time video with addition of an indication of a target location, position of said 
indication being derived at least in part by use of said transformation. 



12. 



The method of claim 1, wherein the endoscope is a bronchoscope. 
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13. A system for ensuring correct alignment of an endoscope during 
performance of an endoscopic procedure, the system comprising: 

(a) an endoscope having a distal end for insertion into a body; 

(b) an imaging arrangement associated with said endoscope and 
configured to generate a real-time video image of a region beyond 
said distal end; and 

(c) a processing system associated with said imaging arrangement and 
configured to: 

(i) in an initial state of alignment with a target location, derive 
from said imaging arrangement a reference image 
corresponding to correct alignment with the target location, 

(ii) derive from said unaging arrangement real-time images of the 
region beyond said distal end, and 

(iii) co-process said reference image and said real-time images to 
detemiine a current aHgnment status of said endoscope with 
the target location. 

14. The system of claim 13, wherein said processing system is configured 
to co-process said reference image and said real-time images to determine a 
measure of mismatch, said processing system further generatmg an alarm signal if 
said measure of mismatch exceeds a predefmed value. 

15. The system of claim 1 3 , further comprising a display for displaying at 
least said real-time images to a user, wherein said processing system is configured 
to co-process said reference image and said real-time images to determine a 
displacement correction required to compensate for erroneous movement of the 
endoscope, said processing system further generating an indication on said display 
indicative to a user of said displacement correction required to compensate for the 
erroneous movement of the endoscope. 
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16. Hie system of claim 13, further comprising a display for displaying at 
least said real-time images to a user, wherein said processing system is configured 
to co-process said reference image and said real-time images to determine a 
transformation relating said real-time video frame to said reference image, said 
processing system further generating on said display an indication of a target 
location, position of said indication being derived at least in part by use of said 
transformation. 

17. The system of claim 13, further comprising a position measiiring 
system including a position sensor carried by an elongated element removably 
deployable along a working chaimel of said endoscope. 

18. A method for facilitating performance of an endoscopic procedure on 
a target which is obscured from view by an imaging arrangement of an endoscope, 
the method comprising: 

(a) generating real-time video from the imaging arrangement of tiie 

endoscope; 

(b) determining a position of the target and a position of a distal end of 
the endoscope; 

(c) generating a display including the real-time video and a simulated 
view of the target correctly positioned within the real-time video; and 

(d) adjusting the display so as to mamtain the target correctly positioned 
within the real-time video when the endoscope is moved. 

19. The metiaod of claim 18, wherein said adjusting includes comparing 
features from at least one frame of the real-time video with features from a 
reference image derived from the imaging arrangement during said step of 
determining, thereby deriving a transformation relating said real-time video frame 
to said reference image. 
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20. The method of claim 18, wherein said determining a position of a 
distal end of the endoscope is performed using position measioring system including 
a position sensor carried by an elongated element removably deployable along a 
working channel of said endoscope. 
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